TanhExp: A smooth activation function with high convergence speed for lightweight neural networks

Lightweight or mobile neural networks used for real-time computer vision tasks contain fewer parameters than normal networks, which lead to a constrained performance. Herein, novel activation function named as Tanh Exponential Activation Function (TanhExp) is proposed can improve the performance these on image classification task significantly. The definition of TanhExp f(x) = x tanh(ex). simplicity, efficiency, and robustness various datasets network models demonstrated outperforms its counterparts in both convergence speed accuracy. Its behaviour also remains stable even with noise added dataset altered. It shown that without increasing size network, capacity lightweight be enhanced by only few training epochs no extra added.